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Application of Judgmental Standard Setting 


Proceduges to Vocational Evaluation Competency 

- 4 Statements by Rehabilitation Field Personnel and Educators , 

‘Methods of standard setting have been described and categorized 
by numerous authors in the fields of educations and psychological 
| measurement (Buck, 1977; Hambleton, Powell, & Eignor, 1979; 
Livingston & Zieky, 1978; Meskauskas, 1976; Popham, 1978). . 
Prevalent in the literature are three peneral classifications of . * 
methods including normative, criterion, and judgmental methods 
(Buck, 1977; Hambleton, Powell, & Eignor, 1979). Normative 
procedures produce standards that have been established as a result 
of comparison with a norm group, j.e., test scores of individuals 
that have already been assessed. The resulting standard then may be wg | 
related to a particular point on iis test score scale, such as one | 
standard deviation below the mean, or at the sixteenth percentile 
(Glass, 1978). Criterion based standards sie those which.. are 
built upon the existence of an outside criterion measure, s 
performance measure, or true ability distribution" (Hambleton, ; 
- Powell, & Eignor, 1979, p. 43). Widamental methods may be employed 
without the use of examinee data to set a standard. Unlike 
normative and criterion methods, decisions are made relative to 
test item content with judgmental procedures where raters have 
access to test items and, not external criterion such as examinee 


data. The most prevalent judgmental procedures for setting 


at 


“standards a are‘those described by Ebel (1972), Anggff F (1971), and 
Nedelsky (1954). : * 


| 


Not unlike other bacinatiions requiring the eurditieatlcit or 


other documentation of minimal competence, the field df vocational 


ea luneion abi developing procedures: for the formal - 


certification of vocational evaluation personnel (Coffey & PAREN» aN 


1980). Actually, the certifisation of vocational evaluation 


personnel has been an issue for a number of year’s (Baker, a ‘ 


| Brubaker, 1979; Coffey, 1978; Coker, 1980; McDaniel. 1978; Stout, 


1969). This movement within he. field of vocational evaluation ‘ed 


by the Vocational Evaluation and Work adjustaant Association 
(VEWAA), a division of the National Rehabilitation Association, : 
began with efforts to determine professional standards for 
vocational evaluators. VEWAA has been involved with standard 


development for at least 10 years. (Couch, 1971). 


\ 
A prerequisite to the savbitication issue and an area of much \ 


concern involved the establishment of certification critepia. 


favored approach .amQN9 the professionals in the field of ‘vocational 


evaluation. has been to determine what competencies are necessary to. 
| function effectively as a vocational evaluator. For example: 
Coffey (1978) identified more than 2500 competency statements that 


related to the work of vocational evaluators from educators, ' 


| practitioners of vocational evaluation, and literature; Sink, 


Porter, Rubin, and Painter (1979) examined differences and 


a 


w 


similarities. of competency requirenents for vocational evaluators ne 
and rehabilitation couriselors ; and the identification and Rar Ae 
’ classification of competencies was senna: critical to the 
prondes of competency-based certification by Gannaway and Sink 
(1979). y er - 
From. the massive list of over 2000 |vocational evaluator 


competency statements’ compiled’ by Coffey (1978), a final list of 


" 175 competencies was extracted through combining, categorizing, and 


omitting ‘statements (Coffey, 1978). hs the profession of vocational 
evaluation and its isaunsnares for certification have moved in the 
direction of formalizing-certificatign Urbcedures, a critical need 
“emerged to apply a systematic procedure to this list in order to 
refine and accurately address the most relevant or coré competencies 
required for the profeséion. With, the Heatetin by me Vocational 
"Evatuation and Work ijustitent Association (VEWAA) to proceed with 
developing ueeieiataen procedures for vocational evaluation and 
_ work adjustment personnel, came the necessity to refine dnd 
prioritize competency requiremegts for the assessment of) minima] 
_-competency to engage. in the work of these professions. This study 
was an attempt to provide additional information relative to core 
competency requirements as perenived by rehabilitation, evaluation © 
educators and field personnel representing the practi¢ing 
professionals in vocational evaluation. The problem addressed in 


this study was the need to refine and prioritize competency areas 


» 


\ 


within: the field of vocational evaluation fof the purpose of sean 


in the development of vocational evaluation professional 
certification procedures. 
| Objectives of the Inquiry Poe } 
, Traditionally, judgmental standard setting methods have been! 


used exclusively at the test item level.. In this study the Ebel. 


. and Angoff methods of st. dard setting were utilized to eerie hit 


minimum competency standard: on a list of 175 identified competency 


statements for vocational evaluators. The following research \ 


questions were addressed: \ 8 


1. Do the EbET and Angoff methods of standard setting Reels 
-in the establishment of different competency standards when siete 


to competency statements for vocational evaluators? 


2. Do vocational evaluation field personnel and vocational: 
. ® r 


evaluation educators establish different competency standards when 


oye these methods ? 


® 


3. Are the competency standards determined by the different 
methods moderated by group membership? ; ( 


Method 
Subjects were 43 raters reprasenkiny two professional groups. 
The first group consisted of 19 wncattonnt evaluation field 


personnel, and the second group was composed of 24 vocational 


evaluatian educators from university settings. The practitioners 


were state chapter presidents of the Vocational Evaluation and Work 


’ \ 


x 


5 
Adjustment Assocfation, and the educators were identified from the 
current membership directory of the National Council on \ 
Rehabilitation Education. | 
Each rater in each group rated ‘each of the 175 competency 
statements utilizing first the Ebel-method followed by: the Angoff 
method . The raters worked fndependently as the instructions, 


persia data sheets, taehng forms , and competency statement he 


were distributed via mail. 


‘For the Ebel ti nine categories were listed on the rating 


form, and the raters were instructed to check one of the nine 
categories adleating whether the competency statement bint raten 


was considered essential, important-easy, important-medium, 


acceptable-easy, acceptable-medium, acceptable-hard, questionable- | 


easy, qdestionable-medium, or questionable-hard: These categories 
made it necessary for the raters to decide upon the relevance of 
> the competency as well as the difficulty level of the, competency 
statement. These decisions were to be made in reference to 
beginning vocational evaluators minimally competent in the 
profession. . 

The second method, Angoff, required raters to estimate the 
percentage of beginning vocational evaluators, minimally competent 
{ij the profession, who in their opinion would be able to demonstrate 


the listed competency statement. s 4 ‘ 


s, 


The Ebel-Angoff ordering of the ratings was prescribed in an 
attempt to minimize contamination of the ratings due to éarryeousi: 
effects. The Ebel procedure required the aise to make general 
nonnumerical ratings, whereas the Angoff procedure mandated that a 
specific number, (a proportion) be assigned to the competency 
statement. _ Poth methods are concerned with competency difficulty, 
but in the Ebel method it is estimated indirectly and is less 
obvious to the raters than in the Angoff procedbxe. ) 

After all competency statements were rated b the two 
(Procedures, the raters opened a separate envelope with instructions 
to provide a percentage representing an expected success rate for 


each of the nine Ebel categories. In essence the raters were asked 


Insert Table 1 about here 


to determine what percentage of the competency statements allocated 
to each of the ine categories should be required to determine 
minimal competency. | ‘ 

To determine if the use of different methods resulted in | 
different competency edendwnds across ‘goups or if different groups 
determined different competency. standards with the sam} method, a 
2x 2 mixed design analysis of variance was performed with group 


membership representing the between group factor and rating method 
‘ ad 


* 


the within group factor. The dependent variable was the minimum 
‘ e 
competency standard obtained from each rater with each method. 


4 Results and Conclusions 


Insert Table 2 about here 
Significant differences, F(1, 41) = 58.62, p < .001, were 
found between rating methods across groups. Use of the Ebel method 
resulted in a mean competency standard (120.75) Stn fica bie 
‘ “higher than that set with the Angoff method (84.09)., 


Insert Table 3 about here 


The groups of raters also set significantly different, 
% ; F(1, 41) = 4.84,.p*< .05, competency standards across methods. The 
mean competency standard established by the field personnel (100.88) 
was significantly higher than the mean competency standard set by 
the educators (97.31). The importante of this determined group 
difference was minimized due to the significant }nteraction between 


rater group and rating method, F(1, 44) = 5.18, p <..05. Here, the 


Insert Figure 1 about here 
' 


difference between the mean competency en by the 


practitioner group and the educator group was greater for the 


E rd 


Angoff method (96.43 - 74.33 = 22.10) than for the Ebel method ges 
(121.34 - 120.29 7.1.05). nao . a 
‘Additionally, interrater, reliability was determined for the 
+ Ebel (r = 97) and Angoff (r= .93) methods along with the 
intercorrelation between the methods (r = .83)." The average | 
ee, S Dy de> 


Insert Table 4 about here 


oe competency ratings for the Ebel and Angoff methods were correlated 
. \, 
with the average ratings of importance of these competencies -_ 


obtained in a previous study by Coffey. The Ebel and Angoff 


Insert Table 5 about, here 


competency ratings correlated .88 and .70, respectively, with the 
importance ratings obtained by Coffey. 
Educational Importance of the Study 


q 


Historically, no precedent has been set in the utilization of 


judgmental standard setting procedures with any data other than . 

test items. Various studies have been conducted demonstrating the 

use 6f these methods with competency examinations (Andrew & Hecht, 

1976; Archambault, Behuniak, & Gable, Note 2; Behuniak, Gable, & 
Archambault, Note 3; Colton & Hecht, Note 4; Koffler, 1980; Poggio, 

Glasnapp, & Eros, Note 5; Sigmon, Halpin, Halpin, & Sides, Note 6; 

Skagun & Kling, 1980). In the present study the judgmental methods , 


4 10 
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developed and described by Ebel (1972) and Angoff (1971) were | 

utilized with competency statements which represents an dnttrely.| 
new and innovative application of these standard setting methods. 
Personal communication with the developer of one of these 
techniques (Ebel, Note 7) reinforced the initiation of this study. 
wkend justification for this annroaeh was provided by Stewart and 
Mas low (Note 1) in the following: ‘ 

Procedures other than written tests do not provide such 

convenient units as ‘test fitems' as a basis for 

standards setting.... Nevertheless, 2 is possible 

to apply to other measures the same concern to . 

standardization, “reliability, and control of the 

judgmental process. (p. 15) 

In this unique application of judgmental standard setting 
procedures to competency statements, fests obtained in the present 
study were similar to the results found with the tyaditional 
application of these methods to test items. As in previously cited 
studies (e.9., Andrew & Hecht, 1976; Skakun & Kling, 1980: Colton & 
Hecht, Note 4) where method differences were found among various 
judgmental standard setting procedures, this study resulted in the 
formulation of the conclusion that the Ebel and Angoff methods may 
be utilized to determine competency standards, but will result in: 


s 


differént competency standards established. . 


. 


cor 


106 
The results of this investigation tend to substantiate the 


statements of Ebel (Note 7) and Stewart and Maslow (Note 1) that 


, $ 
these procedures (standard setting) may be applied to measures 


other than test items. The significant findings in the reliability 
and validity studies within the present investigation also support 
the imsentigt tp contention that item dudonental standard setting 
procedures (i.e, Ebel afd Angoff) can be suc essfully applied to 
competency statements as Well as test items. 


The demonstrated appl#cability of item judgmental standard 


~ setting procedures; i.e., Ebel and Angoff, to measures other than 


test items provided new insights into the versatility of these 
methods. Through the use of these procedures at the competency 
level, the task of devataping tects for competency certification 
became better defined. As test items are developed for each 
competency identified, the decisions to be made relative to overal] 


campetency for a specific profession may be made at a level that is — 


' substantially more specific to the profession than at the test item 


level. 

The analyses performed and results obtained in the present 
{nvestiqatibn of vocational evaluator competency statements may be 
utilized in the establishment: of either noncompefsatory or 


compensatory certification models (Allen & Yen, 1979). 


‘Noncompensatory certification refers to requiring a minimal 


competency level for each of an established number of specific. 


“er 


ac es. 1] 
subareas within a profession. Ina sannecakuny model , only a 
total on level or mastery. level is required without 
requiring mininally expected ‘levels within, subareas,” For example, 
if a university graduate program requires a total Graduate Record 


Examination (GRE) score of 1000 for admission without requiring, 


specific scores for the verbal. or mathmatics sections, the 


Compensatory model is ‘being utilized. However Mt the school ; 
requires a GRE Score of 1000 with minimum scores of 450 on each 
area a leant model is being applied (Allen & Yens 1979). 

| Theoretically, the. methodology prncnnch investigation 
can be utilized in other areas to increase, possibly insure, the 
presence of content validity in future test development for 
professional certification, licensure, or vayietin. 

The application of the results of this study to a certification 


procedure for purposes of standard setting precipitates the decision 


‘of which competency standards to utilize. Consistently, the - 


different methods, Ebel and Angoff, produced different competency 


standards. A conservative approach might be to choose the results 


" produced by the Angoff procedure averaged across the two groups of ‘ 


raters, since the Angoff procedure consistently resulted in lower 
competency cut-offs. However, the educator group set the lowest 
standards utilizing the Angoff method consistently. Another 
approach may be to select the results produced by the Ebel procedure 


averaged across rater groups. In this study, as in others, (Andrew 


‘13 
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—— 12 
b ‘Hecht, 1976; Sigmon, Halpin, Halpin, & Sides, Note 6) the Ebel 
procedure consistently resulted in higher cut-offs, but ‘was also 
the most stable across rater groups. Other. researchers have. 
suggested combining methods or utilizing several methods, to arrive 
at the cut-off decision (Koffler, 1980; Livingston & Zieky, 1978). 
By combining the two methods and averaging across rater outs the/ 
resulting competency standards a.2 based upon decisions made by 


both groups of raters utilizing bot methods. For example, in this 


study the field personnel established competency standards of 96 


competency statements and 121 competency statements via the Angoff 
and Ebel methods, respectively. The average across the -two methods 
is 108 for the field personnel group. Similarly, for the educator 
group competency standards via the Angoff and Ebel procedures were 
74 and 120, respectively, for an average across methods of 97. 
Those two averages could then be combined and averaged for a 
competency standard of 102 of 175 competency statements, which 
results in a cut-off of 58% (Sigmon, Halpin, Halpin, & Sides, Note 
6). ‘This figure could then be utilized for a compensatory model 


for certification procedures. 


\ 
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Table 1 " 


“ Expected Success Rates by Field Personnel and Educators 


SY. ir eer Ne aR curr ae 


Item Expected Success ‘Expected Success 
\ » Category Rate-Field Personnel Rate-Educators 
t , 
etal ea 1° ‘ 87 
Important-Easy . 87 7 .83 
Important-Medium : 76 82 
Acceptable-Easy ok .67 
Acceptable-Medium .65 | 63 
Acceptab1e-Hard | % a7 56 
Quest ionable-Easy . 30 31 
Questionable-Medium wee 325 


Questionable-Hard . 16 | 16 


SS 
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Table 2 


Analysis of Variance for Rater Groups and Methods 


ER eee eT SOE, ee aL Oe Peres 


Source . df } NS. FE 
ee Fe Eee ee eae Te ree ae 
“Rater groups 7 2841.55 4.84* 
‘Error (between group) 41 586.74 a % 
Methods * ; 1 9 2 26629.55 58.62*** 
Rater x Method Interaction 1 2 2351.09 5.18* 
Error (within group) 41 454.29 - 
*p< 05. p< 001 4 


21 
Table 3 
Means by Groups and Methods 
» 
Method Row 

- Group Angoff “Ebel. Means 
Field Personnel 96.43 + | 121.34 100.88 
"Educators . 74.33 120.29 97.31 o 
Column Means . 84.09 120.75 3 

f | , > 


*n = 175 competency statements 
| 


| 


Table 4 
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Interrater Reliability for Each Group by Each Method 


Ebel Angoff 


Field Personnel 
age 
Combined Groups 


2400 , 


Table 5 


Sigmon-Coffey Data Relationships \ : 
Sigmon Group/Method! Coffey Group Rho* 
4 i E 

Comb ined/Angoff . Combined a * 240 

Combined/Ebel Combined 88 

Field Personnel/Angoff Practitioners : 63 

Educators/Angoff Educators . .63 

Field Personnel/Ebel l Practitioners 81 

Educators/Ebel Educators 87 

Field Personnel/Angoff Educators 63 . 
Educators/Angoff | . ‘Practitioners 67 er 
Field Personnel/Ebel Educators -_, .79 7 


Educators/Ebel ‘ _, Practitioners 85 

“All correlations were significant at p< .001 

154 gmon grouping = combined (19 Field Personnel and 24 Educators) 

2coffey grouping = combined (96 Practitioners and 20 Educators) bs 


! 5 ) 
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o tet e ee 
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Figure 1. Mean competency standard for groups by 

P 
method. 


